BO 1tl7 368 

iOIHOfi 
TITLE 9h 



IMSTITOTIOM 
SPONS aGENCI 

BEPORT hO 
PUB OAT£ 
COMTEACl 
NOTE 

A7AILABIE FfiOH 



EDBS PfilCE 
DESC^If ICBS 



D0C08BVT BBSOBE 



TH 006 822 



IDESTIFIEBS 



Be jar, Isaac I.; Aad Others 

in Infonation Coiparison c£ CcnTcnticcal and 
idaptiT€ Tests in the Beasureient c£ Classrooi 
Achieveient. Besearch Beport 11-1. 
Hiniiesota Univ., Hinneapolis. Dept. of Psychology. 
Office of Naval Besearch, Arlington, Va. Personnel 
and Tra^ining Besearch Erograis Office. 
PHP-BB-77-7 
Oct 77 

N0OOH-76-C-0627 
55p4 

Psycbcietric Methods Program, Oepartvent of 
Psychology, Oniversity of Binne^ota, 75 East fiiver 
Bead, flinneapcli^ Binnesota (Free of charge) ; 

National Technical Inforaaticn Service, Springfield, 
Virginia 

fiF-$0.83 HC-S3.50 Plus Postage. 

♦Acadeiic Achieveient; ♦Achicveaect Tests; Biology; 
*Colle9€ Students; ^Cciparative Analysis; Coiparativey 
Testing; Higher Education; Itea Analysis; Itea Banks; \ 
♦Beasureaent Technigues; Bespcnse Style (Tests) ; 
Scoring: ^Standard Error of Beasuxesent; Test 
Iteas 

♦Adaptive Testing; Coaputer Assisted Testing; Test 
Length 



ABSTBACT 

Inforaation provided by typical and iaproved 
conventional classrooa achieveaent tests vas ccapared vith 
inforaation provided by an adaptive test covering the saae subject 
aatter. Both tests vere adainistered to over 7C0 college students in 
a general' biology course. U^ing the saae scoring aethcd, adaptive 
testing vas found to yield substantially aore precise estiaates of 
achieveaent level than the classrooa test throughout the entire range 
of achievaent, e%en in th^ range where the iiprcvcd conventional test 
tfds designed to be optiial. Adaptive testing also aade it possible to 
reduce the length of the test. An analysis of^ the effects of 
expanding an adaptive test itea pool indicated that iaproved 
precision of aeasureaeni; could result froa the addition of only 
slightly aore discriain^ting iteas. A -eoaparisan of response pattern 
inforaation values (observed inforaation) with test ^ofora&ticn 
values (theoretical inforaation) shoved that the observed inforaation 
consistently underestiaated theoretical inforaation, although the 
pattern of results froa the tvc procedures was guite siailar. It was 
concluded that the adaptive aeasureaent of classrooa achieveaent 
results in scores which are less likely to be confounded by errors of 
aeasureaent and, therefore, arc aore likely to reflect the true level 
of achieveaent. (Author/BV) 
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scores derived from the adaptive' -test were more precise even in the range of 
achievement where the improved test was designed to be optimal. An analysis 
of the. effects of expanding an adaptive test item pool indicates that Improved 
precision of measurement can result from the addition to the pool of only 
slightly more discriminating items. A comparison of response pattern 
information values (observed* information) with test information values 
(theoretical information) shows that the observed Information consistently 
underestimates theoretical information, although the pattern of results from 
the two procedures is quite similar. It is concluded that the adaptive 
measurement of classroom achievement results in scores which are less likely to 
be confounded by errors of measurement and, the^-efore, are more likely to 
reflect a testee's true level of achievement. In addition, the reduction in 
number of test items administered by the adaptive measurement of achievement 
can result in additional time spent in instruction. 
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Ar Information Comparison of Conventional and Adaptive 
Tests in the Measurement of Classroom Achievement 



Achievement testing consists of mapping an individual's proficiency level 
onto an observable indicator of proficiency. This mapping is accomplished by 
means of a testing procedure* Two of the characteristics defining a testing 
procedure (Sympson. 1975) are the nature of the items in the test and the way 
in which the test items are administered* Both of these characteristics are 
potentially important factors in determining how accurately the observable in- 
dicator wxll reflect the individual's underlying proficiency level* 

Given ^n item type, there are basically two ways of administering a test 
—individually or in groups* In group testing everyone answers the same set 
of test items; in individualized or adaptive testing everyone receives a 
different set of items, and the difficulty of a test is dynamically tailored 
to the ability level of the testee. The psychometric advantages and disadvan- 
tages of these two modes of administration have been the subject of research 
in recent year^ (Weiss, 1976; Weiss & Betz, 1973). Results of this research 
suggest that adaptive testing is superior to grcup (conventional) testing in 
terms of precision of measurement (McBride & Weiss, 1976; Vale & Weiss, 1975b; 
Weiss, 1976), test-taking motivation (Betz & Weiss, 1976), and potential to 
eliminate bias (Pine & Weiss, 1976). 

Virtually all of this research is based on ability measurement rather 
than achievement measurement. The question which arises, therefore, is whether 
or not similar benefits would accrue in achievement testing. Since achievement 
testing can be conceptualized in several ways (Green, 1S7A), however, a general 
answer to this question may not be possible. For example, mastery testing 
(Block, 1971) is an approach to achievement testing which is currently receiv- 
ing attention from boch practitioners and theoreticians. The purpose of 
mastery testing is to classify individuals into two states: mastery and non- 
mastery. Because of the instructional philosophy behind mastery testing, there 
Is likely to be a lack of variability in performance at the time of testing on 
a given instructional unit; and as a result, it becomes profitable to tailor 
the length of a test rather than its difficulty. Ferguson (1969) has demon- 
strated the feasibility of implementing such a testing system* 

However, when instruction is likely to result in substantial variation 
with respect to achievement in the population being tested, the procedures 
for adaptive ability testing beccme relevant for achievement testing, provided 
that the same response models which apply in ability testing are also appli- 
cable in the measurement of achievenent. In a previous report Bejar, Weiss, 
and Kingsbury (1977) established the plausibility of that assumption in a 
college instru tional setting. The purpose of this study is to investigate in 
that same setting the performance of an adaptive testing model designed for 
ability measurement in comparison to classroom examinations covering the same 
course content. 
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Comparing testing procedures Is difficult (Sympson, 1975) since diff- 
erent procedures usually differ In more than one respect. Comparisons 
between testing procedures are further complicated by the criteria for 
evaluation (Weiss & Betz, 1973) . Reliability and correlational Indices 
have been used to compatjfc testing procedures In many live data Investiga- 
tions (e.g., Betz & Weiss, 1975; Vale & Weiss, l975a) and in some simula- 
tion investigations (e.g., Jensema, 1976, pp. 82-89). Such comparisons are 
less than optimal. By summarizing all the data in one single value, 
important information is likely to be lost (Samejima, 1977). 

A raore appropriate evaluative criterion for comparing testing procedures 
is psychometric information. Unlike reliability and correlational indices, 
information is an index of the precision of measurement at all levels of the 
trait b^ing measured. Information functions are particularly useful in 
comparing test models analytically. Bejar (1975) used information functions 
to compare the dichotomous, graded, and continuous response models; Hambleton 
and Traub (L971) used them to compare several logistic test models. Because 
the comparison was among models in these cases, the use of information func- 
tions was appropriate. 

The comparison of the same model under two modes of administration 
(conventional and adaptive) is of interest :n research on adaptive testing. 
In this research (e.g., McBride & Weiss, 1976; Vale & Weiss, 1975a) informa- 
tion functions have been computed by monte carlo procedures. The relative 
efficiencv ^f the two modr.s of test administration has . then been determined 
by the ratio of the information functions. The results of such comparisons, 
however, are theoretical predictions which should be verified empirically. 

Research comparing conventional and adaptive testing, using information 
as the evaluation criterion, has been based almost exclusively on monte 
carlo simulated data. These simulation studies suggest that adaptive test- 
ing yields more precise scores than conventional testing; they are not ent irel 
generalizable, however, since they are based on data that fit the model 
perfectly. There has been only one study based on data from live J^^tees 
which used information as an evaluative criterion (Brown & Weiss, 1977): 
however, it was a real-.lata simulation study (Weiss & Betz, 1973, pp. 11-12) 
which did not involve the actual adaptive administration of test items to 
testees. 

Purpose 

The major aim of the present investigation was to compare an adaptive 
achievement test to a conventionally-administered classroom test, using 
information as the evaluative criteria. In contrast to previous investiga- 
tions, the measure of information used was derived from live test admin- 
istration of both the adaptive and conventional tests. Because classroom 
examinations are seldom designed to be psychometrically optimal, the adaptive 
test was also compared to an improved conventional test .which was constructed 
from the same item pool. In addition, the data provided an opportunity to 
study the effects of expansion of the adaptive test item pool on its inCorma- 
tton characteristics. 
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Method 

Data for this study were obtained from students enrolled in a large 
introductory Biology course at the University of Minnesota (see Bejar et al, 
1977). Two midquarter examinations and a iinal examination are administered 
in the course. Although each midquarter examination covers several content 
areas, a single dimension has been shown to account for performance on the 
examinations (Bejar et al, 1977). In addition to the classroom examinations, 
volunteers completed two computer-administered adaytive tesE?Vhich covered 
the same content as the midquarter examinations. The data analyzed for each 
student consisted of scores on the two classroom midquarter examinations and 
the corresponding scores on the first and second midquarter adaptive test-:. 
The -results are based on a comparison of the levels of information associated 
with these scores. 



Subjects 

Volunteers were recruited during the fall and winter quarter's of the 
1976-77 academic year. Each quarter an information sheet was distributed to 
all the students in the class which invited them to participate in the 
research project. For participating in the first midquarter adaptive test, 
patticipants received one point which was to be added to their course grade; 
and for participating in the second midquarter adaptive test, they received 
two points. During fall quarter 394 students participated in the first mid- 
quarter auaptive testing and 386 participated in the second midquarter 
' adaptive testing; during winter quarter the corresponding numbers were 317 
and 349, respectively. 

Prooeduve 

For both the first and second midquarter administrations, the volunteer 
students were given three tests in the following order: 1) an adaptive verbal 
ability test, 2) the multiple-choice adaptive biology test based on the 
content covered in the classroom midquarter examinations, and 3) a test con- 
sisting of specially designed biology items. In the present report, only the 
data from the adaptive biology tests were analyzed. 

The three tests were administered by means of cathode ray teruinals 
(CRT) connected to a Hewlett-Packard real-time computer system. Instruc- 
tional screens explaining the operation of the equipment were presented prior 
to testing (DeWitt ^ Weiss, 1974). A proctor was present in the testing 
room at all times to assist students with the equipment. Each test item was 
presented separately at the rate of 960 characters per second on the CRT 
screen. Students responded by pressing the key corresponding to ths ctiosen 
alternative. During the fall quarter administration, feedback was provided 
after each response, i.e., each student was informed whether or not he/she 
had answered each test item correctly. During the winter quarter administra- 
tion, immediate feedback was not provided. There were no time limits Imposed 
on any of the tests. At the completion of testing, students received a 
printed report which listed questions answered incorrectly and provided the 
correct answers. 
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Adaptive Test 



If em pools . The development of the item pools used in this study has 
been described b} Bejar et nl- , 1977. The answer sheets for two midquarter 
examinations from two previous academic quarters were used as raw data for 
obtaining the item, parameters — discrimination (a), difficulty (2?) , and 
guessing ^o) — of the item characteristic curves for the items. From ^ne 
fall quarter administration 114 items were available, which covered the 
contents of the first classroom test; the pool for the second test contained 
112 items, ^ From the winter administration 44 items were added to the first 
test pool, and 49 were added to the second test pool. There was thus a 
total of 158 items In the first test item pool and 161 in the second test 
pool. 

To construct item pools which could be used for administration of 
stradaptive tests (Vale & Weiss, 1975a, b; Weiss, 1973), each of the two pools 
was structured by forming nine strata of increasing difficulty. Mean stratum 
difficulties were chosen so that there would be approximately the same number 
of items per stratum. Within each stratum the items we^^e ordered in terms 
of their discriminations unless it resulted in items covering the same 
content area appearing consecutively • Appendix Tables A and B show the nine 
strata into which the first and second test item pools were structured. 

Effects of expanding the item pool * In a conventional test the distri- 
bution of item parameters will determine the characteristics of scores derived 
from that test. Similarly, in adaptive testing the characteristics of the 
items in the itein pool should influence the characteristics of the scotes. 
The theoretical research on this question (Jensema, 1976, pp, 82-89), 
however, suggests that improving the item pool has little effect on precision 
of measurement. 

The question of improving the item pool in adaptive testing was examined 
by Jensema, using a simulation study with Owen's (1975) BayesLan adaptive 
strategy. Two kinds of pools were studied: one in which a=l,0 and (7=.25 
for all item^ and one in which a=2,0 and c7=,20. The distribution of fe's 
uithin the two pools was the same. Jeasema's conclusion, that improving the it 
pool has no effect on the accuracy of es'timating G, is counter-intuitive. One 
potential problem with Jensema's study is that the dependent variable used 
was the correlation of 6 and 9, which may not be sufficiently sensitive to 
detect cV anges in precision. Furthermore, the composition of the pools used 
by Jensema were atypical, since all the items were assumed to have the same 
discrimination. Consequently, his results lack generaiizability , 

The present data permit a more realistic assessment of the effects of 
item pool characteristics on the precision of adaptive test scores. Speci- 
fically, the response vector information functigps computed f r both adaptive 
t3sts in the winter data were based on an enlarged version of the fall item 
pool. The items that were added to both pools consisted of those items 
administered In the fall classroom test for which it was possible to obtain 
item parameter estimates. 



^Bejar et al. (1077) reported that the second midquarter item pool contained 
123 items; the 112~item pool resulted from the removal of il items which were 
administered ia a special format as the third test. 



Table 1 shows the mean end standard deviation of item parameter estimates 
for both fall Item pools and the same statistics for "he items added to form 
the winter pool. Tor the first test the mean of the added items was 
some%rhat lower than the items in the fall pool. For the second test the 
added items were, bn the average, slightly more discriminating. In terms of 
difficulty, the added items in the f.lrst test pool were, on the average, 
.10 easier. In the second test pool, the added items were only ,02 easier. 
Appendix Tables A and B show that the added items were well distributed across 
the nine strata of the stradaptive test pools. In addition, within strata, 
the new items were well distributed in their order of administration, Average 
stratum discriminations were higher for the improved (winter) pool for only 
three of the nine strata in the Test 1 pool (Table A) and six of the nine 
strata in the Test 2 pool (Table B) . In no case were the differences in mean 
discrimination very large. 



Table 1 



Mean and Standard 
Pool and for Items 


Deviation 
Added to 


of Item Parameter Estimates for Fall Item 
Winter Item Pool for Adaptive Tests 1 and 


2 


Test and Pool 


Number 




a 


b 




0 




Mean 


S.D. 


Mean 


S.D. 


Mean S 


.D. 


Test 1 
















I tarns in^Fall 
















Item Pool 


114 


1.21 


.46 


.18 


1.22 


.25 


.09 


Items Added 
















for Winter 


44 


1.15 


.37 


.08 


1.12 


.30 


.06 


Test 2 
















Items in Fall 
















Item Pool 


112 


1.20 


.40 


.16 


1.16 


.27 


.09 


Items Added 
















for Winter 


49 


1.22 


.40 


.14 


1.23 


.29 


.07 



Implementation , One of the advantages of the stradaptive testing - 
strategy is that prior information can be used to select the stratum from 
which the first item is administered. In this Swudy the entry point was 
selected by the student; at the beginning of each stradaptive test students 
were asked to state their gtade-point-average (GPA) by selecting one Ojf nine 
equally-spaced GPA intervals from 2.00 to 4.00 (DeWitt Weiss, 1974, p. 49). 
On the assumption that overall GPA was related to biology achievement levels, 
students with the highest CPAs began the stradaptive test with an item at the most 
difficult stratum (Stratum 9), while those with the lowest GPA began with an 
item at the least difficult stratum (Stratum 1). 

A variable criterion was used to teminate testing on the stradaptive 
test. After a student answered five items 'in a stratum, if he/she answered 
20% or fewer correctly, testing was terminated. If testing was not terminated 
by this criterion after 50 items had been adminis** red, uo further items 
were administered. 

The branching strategy used in the stradaptive test was: 1) if the 
current item was answered incorrectly or skipped, to administer the next 
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unadministered item from the next easier stratum, or 2) if the current item 
was answered correctly, to administer the next unadministered item from th^ 
next more difficult stvatum. 

Conventional Tepts 

C lassroom tes ts. The classroom examinations each quarter included 55 
items, which the course si^ff selected by a combination of pedagogical 
criteria and proceduref from traditional test theory. Their aim in 
constructing these tests was to produce a "^ood" test for purposes of course 
grading. Students were instructed to answer bO items of their choice. For 
purposes of this research, however, the tests were shorter than 50 items, since 
item parameter estimates were not available for some of/ the Items, 

The item parameter estimates for the items in Fall Tests 1 and 2 (Fl and 
F2) are in Appendix Table C; those for Winter Tescs 1 and 2 (Wl and W2) 
are in Appendix Table D. 

Improved t ests . A major problem in comparing testing procedures is that 
their inherently dissimilar characteristics frequently make equitable 
comparisons difficult. The problem can be alleviated by allowing each strategy 
to function optimally whi ' 3 equating the testing procedures en relevant 
characteristics. The classrooni exams were not expected to be psy<:hometrically 
optimal; therefore, it w?s necessary to compare the stradaptive tests with 
an Improved conventional test drcTi from the same item pool. The winter 
item pool contained all the items available; therefore, only the 
winter data were used in the construction of the iiuproved conventional 
tests, i- 

> 

The improved conventional test was designed to use the most discriminat- 
ing items in the item pool in order to measure individual differences in 
the range of course achievement within which differential grades would be 
assigned. That igi, it was assumed that below a given level of "passing" the 
course, further differentiations among students were unnecessary; above that 
level it was desirable to differentiate as accurately as possible among the 
students in order to assign differential grades. To pemit a psychometrically 
meaningful cpmparison with the adaptive test, the improved conventional tests 
were also designed to be equivalent to the ad5»ptive test in terms of levels 
of item discrimination and number of items administered. 

A comparison of the mean discriminations fot the original winter quarter 
classroom tests with the ite^ pools ytsed foi the stradaptive test chowed that 
the mean for the stradaptive pool was a=l,19 for the Wl item pool and 
for the W2 item pool. Mean discriminations for the winter classroom tests 
were 1,09 and 1,14, respectively. The comparison between the item discrimina-7 
tions of the two testing .strategies is complicated, however, by the way items 
are selected ^'^r administration in the stradaptive test. Since the items in 
each stratum in the stradaptive pool were ordered by ^beir discriminations 
and the branching strategy is designed to' administer the earlier items in the 
strata first, the mean discrimination of the stradaptive item pools will be 
lower than the mean discrimination of items administered in most stradaptive 
tests; 



To provide a fair comparison between the adaptive and the conventional 
testa* it-would be necessary to construct a conventional telit "matching" the 
item discriminations in the adaptive test. This is difficult* to ^plement , 
hovevex:, since the discriminations in each administration of .the adaptive 
test, will differ. Instead, the improved conventional tests were designed to 
provide a comparison which would not favor the adaptive test in terms of 
mean'i-tem discrimination. 

TK^ improved conventional tests for each of the two midquarters were 
constructed by selecting the items which appeared first in the strata of the * 
stradaptive pool; these were generally the most discriminating items in the 
strata. The number of items selected was based on the overall mean test 
length for the stradaptive test. The items which wet»e ordered first in the 
top $even strata of tue stradaptive tests were '^'il'^cted to constitute the 
improved conventional tests. Only seven sf i used rather than nine, 

so that the improved conventional test woi somewhat peaked. Its 

precision would thus be concentrated in the range o£ achievement «iost relevant 
for instructional decisions. The improved conventional tests consisted of 
24 items each for both the first and second tests administered in the winter 
quarter; they were based on a stradaptive test with a maximum of 30 items 
which had a mean test l6nglh cf approximately 2A items. 

The item parameters for the items constituting the two improved conven- 
tional tests are shown in Appendix Table, E. The first 21 items (;pmprise the 
first three items in Strata 3 through 9 for both tests. In the improved 
conventional tests the last four items were the fourth items in Strata 7 
through 9. These it^ms had mean discriAination values of 1.73 an^ 1.76, 
respectively, for the two mid^uarter examinations; for the stradaptive pools 
the mean discriminations were 1.19 and 1,21. 

Because of the way 'the stradaptive i^em pool is structured <ind the way 
stradaptive test itemi:: are selected, the mean discrimination of the improved 
conventional test would be equal to or greater than that of any stradaptive 
test. The mean discrimination of the two testing proc'edures would 'be equal 
colely for a testee whose stradaptive Lest response record included only the 
items in the improved conventional test. For any testae whose responses on 
the stradaptive test ^required administration of items fart;her down the strata 
than those used b> che improved conventional test, the mean discrimination 
would be lower than that of the conventional test. Since the majority of 
stradaptive response records utilize items beyond tne third item in th^ strata, 
the stradaptive tests generally would use items of lower average discrimination 
wOan would the improved conventional test. 

Scoring 

All tests were scored by maximum likelihood estimation, specifying 
Blrnbaum's (]968) three-parameter logistic model as the response model. The 
item parameter estimates were edited by the scoring program so that the 
maximum value of the discrimination parameter (a) was set to 2.5, the maximum 
absolute value of the difficulty parameter (b) wa« set to 3.00, and the 
maximum value of the guessing parameter (c) was set to .35. In estimating 
achievement levels, omitted items were not scored as incorrect; they were 
merely ignored. 
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Information 

Definitions. Equation 1 gives the test infcmatidn function of a test 
consisting of n items, in relation to the logarithm of the likelihood function 
of response pattern V (see Samejima, 1969): 



1(6) 



86^ 



[11 



whjere ^^^O) is the likelihood function, and 



V is the pattern of coprect/incorrect responses to a set of test items. 
That is, information is the (negative) expected value of the second deriva- 
tive of the log likelihood function. "Psychometric information," defined in 
.this way, is^ identical to Fisher'*- concept of information (cf. Edwar4,s, 1971). 

In this study the comparison between the conventional and adaptive tests 
was based on^^observed information functions. These were computed from the 
item responses given by e^ach testee. The observed value of information, as 
opposed to the expected value, is the value of the second derivative of the 
log-likelihood function at a testee 's estimated value of 6. That is, 



1(9) = 



99^ 



(21 



Equation 2 defines the response vector counterpart of Samejima's (1969, Ch. 6) 

dtem response 'information function which she has called the response pattern 
information function (Samejima, 1973). 

For the 3-parameter-logistic model, 1(6) is giveu by 
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where = 1. 7 

7 = the estimate of the discriminating power of the Item 

n 

- = the estimate of the lower asymptote of the item characteristic curve 

P 

X * na iO-h ) 
G . 

I = the estimate of the difficulty of the item 

a 

_ jl if item is answered correctly 
~ In if item is answered incorrectly. 



It is clear from Equation 3 that for a single item, 1(9) takes one of two 
values, namelv 



r^(0) 



~ , if u = 1 

a 
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1^(8) 
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Equation A occur? with probability P « P^(G) = + (l-o^) [l+e'^^r\ while 
Equation 5 occu- s with probability Q = 5 (9) = 1 - P (6)^. Thus the expected 



value of I (6) (i.e., I (9)), is 



A 

1^(6) 



(1-P) 2_ 



2___ + p -J± r 



[6] 
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which is the usual item information function evaluated at 6 (see Birnbaum, 
1968, Eq. 20.4.20). The sum of the I^(§) across all items administered in 
a test at a given value of 9 Is 1(9), which is the theoretical test iili forma- 
tion function based on estimated values of 9. Brown and'Weiss (1977)u8ed the 
evaluation and summation of item response information functions by Equation b 
at an estimated value of 9 in their Ifve-data simulation stufly to oDtain 
estimated information curves; their g's, however, were ba^ed on a Bayesian 
scoring routine. 

Both 5"(9) and 1(9) depend on the item parameter estimates a, b, and a. 
However, 1(9) is one step further removed from the data, since it does not 
allow the observed response pattern of correct and incorrect response to 
dictate its value, whereas 1(6) does. In theory ?(0) may be considered an 
estimate of 1(0), which is easily obtained during the estimation of 8 by the 
Newton-Raphson procedure, requiring both the first and second derivative ot 
the log-likelihood function. The value of the second derivative of the 
log-likelihood function at the last iteration is 1(6). 

Computation . Using the maximum likelihood scores computed for each 
testae on the conventional and adaptive tests, information was computed 
for each testee during the scoring process by evaluating the second derivative 
of the log-likelihood function at the final estimated value of 9, based on 
test items actually administered. The response vector information curves 
for a given testing strategy weie then obtained by grouping students on their 
estimated achievement (§) in intervals of .20 from -2.00 to 1-2.00. The mean 
response vector information for students within a given interval of 9 was 
assigned to the midpoint of that interval. All information values presented 
below have been multiplied by 1/2.89. 

r omparisoK . No studies have been reported which utilized the item 
response pattern information function [?(§)] computed from live-testing data; 
therefore, it was appropriate to compare the results of computing information 
by this method with the information curves derived from the sum of the 
item information functions. Tlie computation of test information curves from 
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Figure 1 

Observed and Theoretical Test Int«rmation Functions for Test 
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Observf J and Thewretical Test Information Functions for Test F2 
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the sum of Item Information curves assumes that real testees respond to Items 
In the test In accordance with the Item characteristic curve (ICC) model. 
On the other hand, computing Information curves using Equation 3 from the 
Item response pattern of real testees will likely Include some error, since 
all testees do not respond strictly In accordance with the model. A compar- 
ison of the two Information curves derived from the same set of Item responses 
was, therefore, useful to evaluate the applied usefulness of response pattern 
Information functions, as well as to Indicate whether or not the responses of 
the students to this achievement test were widely discrepant from the ICC 
model. I 

Consequently, test Information curves were computed from the sum of the 
Item Information functions (Equation 6) and from the response pattern Informa- 
tion functions (Equation 3), using the responses to the conventional test for 
each of the four mldquarter examinations. 



Results 

Test Information Versus Response Pattern Information 

Figures 1 through 4 show for the four classroom biology examinations 
the test Information curves computed from 1) the svim of the Item Information, 
I.e., the theoretical test Information function [J(6)]; and 2) the response 
pattern Information curves. I.e., the ^)bserved test information function 
[?(§)]. Data for the test Information functions are in Appendix Table F; 
data for the response pattern information functions are in Tables 2 and 3. 

The data for fall Quarter (Figures 1 and 2) show that item response 
pattern Information [?(B)] consistently underestimated the theoretical 
curve derived by sunmiing the item information functions [/(§)]. The 
difference was fairly consistent throughout the 9 range, although for the 
first test (Fl), the discrepancy diminished at the lower end of the 6 
continuum, where G'£ -1.40. For both sets of data the largest differences 
appeared to be at the point of highest information; the magnitude of 
differences decreased with decreasing information levels. 

^ The winter data (Figures 3 and 4) exhibited the game general pattern of 
results. It can be seen from Figures 3 and 4 that t(6) again underestimated 
the value of the theoretical test information function. In the first test 
(Wl) there "as a marked decrease in the discrepancy between the two curves 
for those v xues of 9 less than about -].25; in the second test (W2) data the 
two curves were closest together at values of 9 less than 1.50. The v^inter 
data, however, did not fully support the tendency for the two curves to be 
farthest apart at the point of highest information; this tendency occurred 
in the W2 data, but not the v^l data. 

There were thus three crends common across all four examinations: 

1. The observed (response pattern) curve was alway.^ an underestimate 
of the theoretical (test) information curve; 

2. The differences between the two Information curves tended to 
diminish, and in sone cases disappear, at low levels of 6; and 

3. There was a fairly constant difference between the two information 
curves throughout the range of 9 above -1.00. 
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Figure 3 

Observed and Theoretical Test Information Functions for Test Wl 
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Figure A 

Observed and Tlieoretical Test Information Functions for Test W2 
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Adoptive Test Versus Claaavoom Teat 

Fi vat teata * Table 2 shows the values of observed information 
(respcr^** pattern information) from the first classroom and adaptive tests 
for fall (Fl) and winter (Wl) . The results are plotted in Figures 5 and 
6, which show that for both fall and winter the adaptive test yielded a 
substantia lly^higher amount of information at all levels of achievement 
greater than 0=-1.5, Because the adaptive test was shorter, on the average, 
this is particularly significant. 

Figure 5 

Observed Information Functions for Fl Classroom 
and Adaptive Tests 




Estimated Achieve^ient Level 



As previously indicated, tne number of items was not fixed for the 
adaptive test. Althcuj?h the maxiinuia test length for the adaptive test was 
50 items. Table 2 shows mat on the average students were terminated 
after 27.2 ite^is in Test Fl and ^fter 31.6 items in Test Wl. Excluding 
students at the extremes of the 9\ distr ibut ion (where the stradaptive test 
would tend to terminate premature!l^y because suitable items were not available 
in the pool), the range of mean nu^nber of items to termination on the 
stradaptive test was 18.9 to 32.5 ^or Test Fl and 25,9 to 41. 1 for Test Wl. 
On the other hand, the mean information values for the classroom test were 
based on an average of 35 items for Test Fl and 40.5 items for Test Wl. 
(Although the actual classroom test was 50 items long, there were items 
for which no item parameter estimates were available.) 

Thus, even though the adaptive test on the average was about eight 
items shorter, it yielded a much more precise estimate of achievement. 
Fl . example, at 9=.7^(and .9) the Fl classroom test had maximum information 
of 2.90, whereas at 9=.7 the adaptive test had maximum information of 5.07, 

Vj 
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Table 2 

Number of Testees, Mean Number of Items, and Mean Observed Information (1(6)3 
Intervals of 6 for First Adaptive and Classroom Tests from Fall and Wintet Quarters 



e 

Midpoint 



-1.9 
-1.7 
-1.5 
-1.3 
-1.1 
-.9 
-.7 
-.5 
-.3 
-.1 
.1 
.3 
.5 
.7 
.9 
1.1 
1.3 
1.5 
1.7 
1.9 

Total 
Mean 



Fall Test 1 (Fl) 



Winter Test 1 (Wl) 



No. Testees No. Items Information 
Adap Class Adap Class Adap Class 



5 
8 
13 
15 
14 
16 
18 
19 
33 
22 
31 
32 
21 
16 
17 
17 
12 
10 
10 
1 



5 
24 
26 
47 
41 
60 
53 
63 
81 
64 
68 
83 
68 
88 
61 
59 
49 
40 
44 
20 



16.8 
26 ..6 
26.3 
30.8 
24.5 
31.9 
31.4 
28.6 
22.2 
23.5 
25.5 
29.0 
32.5 
31.3 
30.0 
29.1 
22.6 
18.9 
29.2 
10.0 
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34.8 

35.1 

35.0 
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35.3 
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35.3 

35.2 



No. Testees No. Items Information 
Adap Class Adap Class Adap Class 
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Figur/iB 6 

Observed Information Functions for Wl Classroom 
and Adaptive Tests 




The ratio of information values at 6=.7 was 1.75. This means that for the 
conventional test to be equal in precision to the adaptive test at that 
level of 0, it would have to be increased in length by about 75%. This 
would result in a conventional test of 61 items in order co achieve the 
same quality of measurement as a stradaptive test with a mean of 31.3 items 

^ The stradaptive test achieved its highest level of information (5.77) 
at 6=1.1 with an ayerage of 29.1 items; the Information provided by the 
classroom test^at 9=1.1 was 2.63. The ratio of 2.19 indicates that at 
this level of 9 the classroom test would require 7Z items to measure as 
well as' the 29.1-'item average adaptive test. 

Similar results were observed for the Wl data. At the point where 
the classroom test was most infon. ' 've, §=-1.3, the adaptive test was 
more informative by a factor of 4. 3.55=1.31 with, on the average, 6.4 
fewer items. Thus, at that level of 0 the classroom test would require 
53 items to measure, as precisely as the average 34-item stradaptive test. 
At the point- where the adaptive test was most informative, 9*1.1, the 
improvement factor was 6.74/2.12=3.18. The classroom test, therefore , 
would require 129 items to measure as precisely as the 37-itein adaptive 
test. Thus, even when comparisons were made at the point of maximum 
Information for the classroom test, the adaptive test was more efficient 
in -terms of information per item. When the comparison was made at the 
point of maximum information for the adaptive test, the discrepancy In 
efficiency for the two testing procedures was even greater. 
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Number of Testees, Mean Number of Items.and Mean Observed I^omatlon [!(§)] 
for Intervals of § for Second Adaptive and Classroom Tests from Fall and Winter Quarters 
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Second teate. Table 3 shows the number of testees, the mean number 
of Items, 3nd the mean information as a function of 9 for the second 
classroom and adaptive tests administered during fall (F2) and winter 
(W2). Estimated information curves are plotted for these tests in 
Figures 7 and 8. Figure 7 shows that for the F2 data, the adaptive 
test was generally superior to the classroom test. In the interval from 
9— .50 to 6-. 20, however, the classroom test yielded higher levels of 
information. 

Figure 7 

Observed Information Functions for F2 Classroom 
and Adaptive Tests 
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Figure 8 'shows the results for W2. For all 6 values greater than 
-1.5, the information provided by the adaptive test was substantially 
higher than that of the classroom test; this was. .similar to the findings 
for Fl and Wl. The adaptive test thus provided better measurement through- 
out the § range in three of the four tests. 

There are two explanations for the adaptive test providing less 
information than the conventional test for the F2 data in a narrow range 
around the mean of the 9 distribution. First, as Appendix Table C shows, 
the F2 classroom test was a considerably more peaked test than fhe Ji, 
Wl, and W2 classroom tests. Peaked tests tend to have peaked information 
functions (Lord, 1970), since they concentrate all their measurement 
efficiency near one point on the 6 continuum. 

A mote important explanation, however, is seen in Table 3. In the 
ranee of §-.5 to .10, the mean adaptive test length was Substantially 
below the mean classroom test length. Dividing the ^'^1°^'^.^°^. .1\IT^ 
of these values of 9 by their corresponding test length indicates that 
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Figure 8 

Observed Information Functions for W2 Classroom 
and Adaptive Tests 
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the mean information per item was higjjer for the adaptive test than for 
the classroom test. For example, at 0=-.5 the mean Information per item 
was .11 for the adaptive test and .09 for the classroom test. Thus, while 
observed mean information, was lower for the adaptive F2 data, this was 
merely an artifact and was attributable to the termination rule employed 
in the test, which resulted in very short tests in the 9 range of -.5 to 
.20. 

Surmavy . The results from both test administrations show that when 
d«ifferences in test length were taken into account, the cHaptive tests 
yielded substantially more precise estimates of achievement than any of 
the conventional tests at all levels of achievement. The results, 
summarized in Table 4, were equally favorable to adaptive testing when 
all e levels were combined. As shown in Table 4, the information across 
levelb of e for the Fl data was 4.53 for the adaptive test and 2.36 for 
the classroom test with test lengths of 27.2 and 35.0 items, respectively. 
The information ratio of 1.92 in favor of t^e adaptive test Implies that 
the classroom test would require 67 items in order to measure as precisely 
as the average 27-ltem adaptive test. The results for the other thr^ 
tests (Wl, F2, and W2) also showed the overall superiority of the adaptive 
test while reducing test length. The smallest improvement was for the F2 
data; the ratio of mean information for the F2 test was 1.24 in favor of 
the adaptive test, implying that the conyennional test would require 46 
items to measure as well as an average 32-ltem adaptive test. This 
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Table A 

Mean Information and Mean Test Length for Fali 
and Winter Adaptive and Clasaroom Teats 



Mean Information Me an Test Length 



Test 




Class. 


Ratio^ 


Adap. 


Class. 


Difference 


Fl 


4.53 


2.36 


1.92 


27.2 


35.0 


7.8 


Wl 


5.28 


2.89 


1.83 


31.6 


40.5 


8.9 


r2 


3.79 


3.06 


1.24 • 


31.7 


37.3 


5.6 


W2 


A. 64 


3.03 


1.53 


32.0 


- 40.3 


8.3 



Adaptive divided by Classroom 
Classroom minus Adaptive 



represents a reduction of 30% in classroom test length attributable to 
adaptive testing, while achieving equivalent average precision with the 
peaked classroom test. 

Adaptive Versus Improved Conventional Test 

Test WU Since the improved conventional test was not actually 
administered, it was not possible to compute its response vector information 
function. Instead, mean values of the test (theoretical) information 
function were computed at 20 levels of 9 using Equation 6. The obtained 
values are in Appendix Table G, which also shows the mean values of 
response pattern information^ for the adaptive Wl test, rescored using 
maximum test lengths^ of 40, 30, and 20 items. Base^ on the data in 
Table G, Figure 9 shows the corresponding response pattern Information 
curves for the stradaptive test at 20- and 30-item maximum lengths and the 
test information curves for the improved conventional test. 

As ^gurtf 9 shows, test information for the improved conventional test 
was very low for the low levels of achievement. Since there were no items 
in this tes^ with difficulties less than Z>=-.65, this was to be expecced. 
The significant ct)mparison between the two testing strategies is for 9 . 
values greater than approximately -.40, as indicated i>y the vertical dashed 
line in Figure ^. Within this range, both the adaptive tests had maxir * 
infoiTvation at §=1.10, while the Information curve for the improved co.^ventional 
test \^s almost at its pi>ak. The mean response vector information for the 
20-item maximum length adaptive test -at §=1.10 was 4.78; the corresponding 
value of information for the improved conventional test was 4.52. This 
represents a 6% increase in information, with an average decrease of 5 items. 
A more significant comparison can be made with the 30-item maximu'.*? adaptive 
te&w, since, on the average, it was 24 items long and therefore the same 
length as the improved conventional test. Throughout the range of §, as 
well as in the range in which the improved conventional test was designed to 
function optimally, the value of response vector information for the 30-item 
maximum ardaptive test was substantially higher than that/for the improved 
conventional test (see Figur- 9). Specifically, at §«1.3, where the improved 
conventional test had the hi^' est information, the 30-item mar^'mum adaptive 
test h-''* at least 7% mors Inf oraal ion. At th^it specific value of §, the 
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Figure 9- 

Mean Information as a Function of Estimated Achievement 
Level for Improved Conventional Test and Adaptive Test 
at Two Test Lengths for Wl Tests 
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mean test length Tor the stradaptive test was 22.8 items, or 1.2 items shorter 
than the improved conventional test. The improved conventional test would 
then require 25.7 items in order to measure as precisely as the average 
22.8-item adaptive test. Thus, with test length and average item discriiainations 
equal, the adaptive process resulted in measurement of higher precision. 

Test W2 . Appendix Table H shows the values of the theoretical test 
information functions for the improved conventional test, as well as the 
mean values of response vector inf ortttation for the adaptive test rescored 
with maximum lengths of 40,- 30, and 20 items. The information curves based 
pn these data for the conventional test and for the 20- and 30-item adaptive 
test are plotted in Figure 10. 

The information for the improved conventional test was again very low 
for 9<-1.00 (see Figure 10), because of the way in which items were selected; 
the lowest difficulty level for an item in the conventional test was ib=-,. 61, 
For 9 values in the range providing an equitable comparison of the two testing 
procedures, the information values for the improved conventional test were 
higher than those for the acfaptive test with a maximum length of 20 items, 
for e>.20. However, the mean number of items for the adaptive test at these 
levels of 9 was always less than 20, or four to eight items shorter than 
that of the improved conventional test. 
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Figure 10 

Mean Information as a Function of Estimated Achievement 
Level for Improved Conventional Test and Adaptive Test 
at Two Test Lengths for W2 Tests 
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The comparison of the information curves for the 24-item conventional 
test with that of the maximum 30-item adaptive test provided a comparison 
of the two testing procedures which is equated for mean number o,f items, 
since under these conditions an average of 24^iteir was administered in 
the adaptive test. In the relevant range of §, the adaptive test provided 
generally higher levels of information, except at 0=.3 and 9 =1.1, where 
informatioT. provided by the conventional test was slightly higher, and at 
i"1.3,where both testing procedures provided equal levels of information 
(see Figure 10). The adaptive test administered two fewer items, on the 
average, at §«.3 than the improved conventional test; at the other two 
values of § the number of items administered was the same. 

Simnary . The comparisons between the improved conventional tests and 
the adaptive tests showed that 1) improved adaptive, tests prodded higher 
levels of information with fewer items than the conventional tist and 
2) adaptive tests provided generally -higher levels of information with 
approximately the same mean number of items. The comparisons were based 
on tests with comparable values of item discrimination*^, although the 
discriminations in the improved conventional test were generally higher th 
the mean discriminations of the adaptive tests. One additional factor 
further influenced these comparisons in favor of the conventional test: 



data on which the comHarisons were based were the theoretical information 
functions for the conventional test and the observed (response vector) 
information functions for tlm adaptive test. As Figures 1 to 4 show, 
the theoretical inf ormatior/values consistently over-eatimatad the observed 
information values. Thus, the information values for the conventional tests 
, are probably higher than they would be had they been computed from the 
Response vectors of actual testees. As a result, it can be <:6ncluded that 
when adaptive and conventional tests are matched in terms of test length 
and average item discriminations, the adaptive test results in consistently 
higher levels of information- The improvement in precision resulting 
from adaptive testing le a function of the process of selecting test items 
appropriately matched to the testee's estimated level of achievement, 

Effeat of Expanding the Item Pool 

First teats. The response vector information curves for '^ests Fl 
and Wl are in Figures 1 and 3, respectively; mean information values are 
in Table 2, As Table 2 shows, however, mean test lengths, as well as 
mean information for the two tests differed. Both mean test length and 
mean information were higher for the Wl tests which utilized the enlarged 
item pool. Consequently, a direct comparison between *he two information 
curves would be confounded by test length. 
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Figure 11 

Mean Information Divided by Mean Number of Items 
for Fall and Winter Adaptive Tests Using 
Test I Item Pools 
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To determine whether or not there had been an Improvement In Information 
beyond that attributable to Increased test length, the mean response pattern 
Information at each level of 6 was divided by the corresponding mean test 
length. These data are shown In Appendix Table I and the resulting curves 
are plotted In Figure .1. The two curves differed very little until the 
point at which 9-. 10. Thereafter and until the point at which §*.70, the 
winter data provided slightly more Information. After this point the winter 
pool failed to provide levels of Information as high as those of the fall 
pool. In terms of overall information, however, there was no Increase In 
mean information from fall to winter. 

Se cond te.^ts . Mean values of response vector information for the 
fall and winter are* shown in Table 3, and information curves are plotted 
in Figures 2 and 4. Figure 12 shows the two information curves equated 
for mean test length at each intei^val of 6; numerical values are in 
Append l-x Table I. 

Figure 12 

Mean Information Divided by Mean Number of Items 
for Fall and Winler Adaptive Tests Using 
g Test 2 Item Pools 
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The winter pool provided higher levels of information than the 
fall pool at almost levels of § (see Figure 12). The differences were 
particularly large in the interval 9=-. 5 to §=1.10. The mean response 
vector inforntation values equated for test length across^ all levels of 
g for fall and winter were .12 and .15, respectively; their ratio was 
1.25, which represented a 25% increase in information attributable to the 
expanded item pool with test length held constant. 
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Summary . These results show the expected outcome, That is* the 
Improvement in precision of r jsurement as a function of enlarging the item 
XDol depends ■)n the nature of the items added to the pool. For the first 
tests, the additional Items were slightly less discriminating than the items 
already in the pool; therefore, using the enlarged winter quarter pool did 
not provide precision of measurement which was appreciably better. For 
the second tests, however, the items added to form the winter pool were, 
on the average, slightly more discriminating than the items already in the 
pool. Scores derived from the enlarged winter item pool were thus more 
precise than those from fall. 

Swm avif and Conclusions 

This report compatea the information provided by typical classroom 
achievement tests and improved conventional tests with levels of information 
provided by adaptive achievement tes^s measuring the same course material. ^ 
The evaluation criterion was response pattern information, a measure of 
information which can be used with data obtained from live test administration. 
A comparison of results from the computation of response pattern information 
with theoretical test information indicated that tha response pattern 
information levels were consistently lower than the test information levels 
for a given set of items. Presumably, this was because testees were not 
responding exactly as predicted by the item characteristic curve model. 
However, the shapes of the information curves provided by the two methods 
of computing information were very sim^ilar. This suggests that response 
pattern information is useful as a substitute for the theoretical test 
information function; it is :jasily computed as par^of the maximum 
likeiiaood scoring procedure, and it reflects the cl^^acteristics of live , 
testing data (a char-icteristic which is useful in empirical research). 

As expected, the adaptive testing of classroom achievement yielded 
substantially more precii^e estimates of achievement than the conventional 
classroom achievement tests. This improvement was evident in several 
tests; it was reflected globally, as well as at all levels of achievement, 
when test length «as taken into account. However, the results indicated 
that the degree of improvement of the adaptive test over the conventional 
classroom test depended upon the psychometric characte'-istics of the 
conventional test. For example, the comparison of the Fl classroom test 
with the Fl adaptive test showed a large advantage in favor of the adaptive 
test, since the items in the classroom test were well distributed through- 
out the range of achievement. Qn the other hand, at some levels of 6 
the F2 classroom test provided higher levels of information than the 
stradaptive test. In terms of mean information per itein, however, the 
stradaptive test was still superior to the classroom test. 

This finding serves to Illustrate the possibility that within a 
restricted range of 9, a conventional test can provide higher levels of 
information than an adaptive test unless certain precautions are taken 
in the administration of the adaptive test. One such precaution would 
be not to administer toe few items. That is, in some circumstanc-es the 
termination rule used In the stradaptiv^e test should be modified to insure 
administration of a minimum number of items. A better solutic^n, however, 
would be to continue testing until a re-specified level of information is 
reached for every Individual (Samejima, 1977). A positive byproduct of this 
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solution would be to insure a high and horizontal information function for the ' 
adaptive test. I.e., equal precision of measurement at all levels of achievement. 

On the other hand, these data also reflect the dilemma encountered in 
the construction of fixed-length conventional tests. Such tests can be 
peaked so that' the item^dif f iculties are concentrated in a given region of 6; 
the result will be a test providing high levels of information in a 
restricted range of 6 and low levels elsewhere.. Alternatively, the fixed 
number of items can be distributed in difficulty over the range of 6 
(as in the Fl test used in this study); the result is a horizontal, but 
low, information function. The test constructor cannot construct a 
conventional achievement test with an information function which is both 
high and flat, unless an inordinate number of test items is administered. 
Adaptive testing, however, provides a ready solution to this problem, 
which is confronted whenever there is considerable variability among students 
in degrees of achievement resulting from instm tion. 

Because the classroom tests had n^t been constructed to be psycho- 
metrically optimal, the information provided by the stradaptive tests 
was compared to that provided by J-^proved conventional tests which were 
derived from the stradaptive test, item pools. The Improved conventional 
tests consisted of Items with discriminations at least as high as, and 
typically higher than, those in che adaptive tests and were the same length 
as the adaptive tests. No response pattern information function was 
associated with the improved conventional test, since it had not actually 
been administered. Yhe test information function associated with the 
improved conventional test was, therefore, compared to the response pattern 
information function associated with the stradaptive test, at maximum 
test lengths of 30 and 20 items. Results indicated that the adaptive test 
yielded generally higher levels of information than the improved 
conven t ional tes t . 

These findings indicate that adaptive testing not only was superior 
to typical achievement classroom tests, but also was superior to a conventional 
test which vas designed to make best use of the same item pool to measure . 
individual differences in achievement levels within a specified range. 
The adaptive test both provided scores of higher precision and reduced the 
number of items administered. The conclusion derived from comparison with 
the improved conventional test is conservative, since response vector 
information in the present data consistently underestimated test information. 
In other words, had the improved conventional test been administered and 
its response pattern Information computed, the adaptive tasc with a maximum 
length of 20 it ?ms would, in all probability, have been found to be 
substantially more informative. 

Contrary to previous research (Jensema, 1975). it was found that an 
expanded item pool can improve the precision of measurement of scores derived 
from it by adaptive testing. Jensema' s findings were based on a situation 
in which the items added to the pool were identical, with respect to all 
three ICC parameters, to the items already in the pool. The results of 
the present study indicate that even when the added items were only slightly 
more discriminating, the addition of new items to the adaptive testing pool 
had a fairly substantial effect, globally, as well as at most levels of 
achievement, on the precision of measurement of scores derived from the pool. 
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This investigation has thus shown adaptive achievement teeting to 
be a feasible approach to the measurement of achievement; compared to 
conventional tests, adaptive achl<;vement testing yields considerably 
more precis^ estimates of achie'^ement , even when conventional tests are 
designed to take maximum advantage of the items in the pool. In order 
to exploit the advantages of adaptive achievement testing to its fu^ldst, 
however, it will be necessary to build a closer psychometric interface 
between instruction and testing. Reduction in testing time by means of 
adaptive testing is meaningless if the result- is solely early dismissal 
from examinations. Rather, what is needed is to link adaptive testing 
with an adaptive instructional context, so that reductions In testing time 
can be used in increased instructional activity. 

Atkinson (1976) has described several examples of adaptive computer- 
based instruction. These systems are adaptive not only because they 
sequence instruction differently for each student, but also because they 
differentially allot instructional time to students in order to maximize 
specified objectives. Differentially allotting Instructional time wlll^ 
in all probability, preserve individual differences in achievement. 

This approach to testing and instruction contrasts with the current 
emphasis on mastery learning and testing (Block, 1971). Mastery testing, 
along with related approaches, is based on the conception that if instructional 
time is long enough, every student will attain the same degree of achievement. 
Although this may be true in principle, an increasing amount of research 
suggests that individual differences persevere even when instructional 
time is allowed to vary (Cronbach & Snow, 1977), ^ The implications for ^ 
instruction and measurement are obvious: An unequiv6cally useful system of 
adaptive instruction and achievement testing must be ab]e to consider 
individual differences rather than atte'mpt to create student homogeneity. 
It seems that adaptive testing can meet that challenge. 
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1.00 


-.97 


.35 


3419 


1.23 


1. 4B 


. 25 


3412 


1 


. 12 


1 Q 




jZo / " 


1 . 02 


-1.22 


- .23 


3421 


1 17 


1.15 


. 35 


3051 


1 


■7 Q 


. 21 


. 28 


3257 


. 98 


-1 . 02 


.25 


3436* 


1.12 


1.59 


.35 


32 79* 




.99 


.01 


.28 


3070* 


.95 


-1.28 


!22 


3271* 


95 


1.-32 


.30 


3403 




.99 


.18 


.19 


3036 


.92 


-l.*I8 


.16 


3061* 


. 95 


1.57 


.30 


3069* 




.88 


-.01 


.35 


3014 


.86 


-1.24 


.14 


3427 


. 92 


1.51 


.26 


3211 




.88 


.01 


.13 


3060* 


.86 


-1.31 


.29 


3449* 


91 


1.26 


14 


3002 




.82 


.13 


.14 


3274* 


.85 


-1.05 


.26 


3063* 


.91 


1.51 


.35 


3426 




.68 


.07 


.22 


3238 


.82 


♦1.06 


.21 


3074* 


.84 


1. 79 


.35 


3423 




.66 


.16 


.27 


3032 


.77 


-1.06 


.27 


3420 


.6a 


1.62 


.35 


Mean (F) 


J 


. i i 


1 1 


. 22 


nean v r / 


1 1 A 


-1 . 10 


. 26 


Mean (F) 


1.29 


1 .46 


. 26 


Mean* ^W) 






09 * 


. 24 


Mash* f 

ne an " ' w / 


1 . 11 


— 1 . A J 


. 26 


Mean* (W) 


1.19 


1.47 


. 2 7 


Stratutn 4 










Stra:ura 1 








Stratum 7 








(Y3 Ttems) 










07 items) 




-1.39 


.20 


(20 Items) 








32 56 


2 


31 


-. 33 


.26 


3077* 


2.50 


3408 


2. 50 


i 


.31 


3430 


1 


.13 


•-.30 


.29 


3027 


1.67 


-1.38 


.35 


3437 


1.95 


66 


.28 


3031 


1 


47 


-.33 


.35 


3443* 


1.07 


-1.64 


.35 


3258 


1.24 


81 


.35 


32 54 


3 


. 38 


-.17 


22 


3249 


.91 


-l.f 9 


.17 


3432 


1.72 


.67 


35 


3237 


1 


. 54 


-.37 


.18 


3428 


.90 


-1.56 


.35 


3048 


1.35 


.66 


.33 


3404 




.65 


-.29 


.35 


30/3* 


1.43 


-1.57 


.31 


3413 


1 .40 


. 76 


.35 


3244 


1 


.35 


-.44 


^.2 3 


3205 


1.25 


-1.53 


.19 


3448* 


1.40 


. /3 


.30 


3058* 


1 


.05 


-.43 


'.35 


3078* 


1 24 


-1.65 


. 35 


3439* 


] . J6 


.64 


.32 


3240 




9B 


-.28 


. 1 5 


3057* 


20 


-1.35 


. 26 


3219 


1.23 


-62 


.21 


3268* 




97 


-.26 


.18 


3065* 


1.17 


-1 . 66 


. 3 5 


3072* 


1.02 


6> 


.32 


3208 




.76 


-.u 


. 12 


3235 


1 .15 


-1 .40 


.28 


3277* »Xl 00 


1.04 


. 


3006 




. 77 


- )7 


.33 


3029 


1.13 


-1.50 


. 28 


3035 


90 


68 


. 28 


3259 




. 69 


- . ♦ 1 


. 20 


3201 


J . 07 


-1.34 


23 


3435 


1. 


86 


30 








31 




3008 


.96 


-1.75 


.18 


3447* 


1 18 


.93 


32 


Mean (F) 




» 7 


. ? "> 


3252 


.79 


-1.77 


.35 


30^.4* 


. 94 


.86 


. J 4 


Mean*(W) 


1 


^ \ 


-.32 


.'2 "5 


3003 


.96 


-1.76 


.34 


3230 


.90 


87 


.35 


Str.-»rum 3 










3044 


.87 


-1.42 


.15 


3444* 


88 


.78 


35 


(19 Items) 










- (F) 


1 06 


-1.55 


.26 


3012 


7<; 


80 


.35 


3021 


1 


.96 


^ 40 


21 


' m*(V^ 


1.19 


-1.55 


,2ft 


3260 


.71 


. 84 


28 


3217 


1 


.Of' 


~ 48 


. 14 








305f>* 


.7' 


89 


2 6 


30S2 


1 


.71 


-.93 


00 










Mean (F) 


L. 28 


78 


il 


3055* 


1 


. 71 




.24 










Mean*(W) 


1 22 


. 7^ 


. n 





















ERIC 



Nuke . Itt'ms with asterisks <irp thnne whi<h were added to thf^ pool Winter quartfcr. ^AU 

orMpr itenm w^t in thr podl both Fail M\d Wintrr qu.irtt-^s. 



I 



Table B 

Item Number, Dltcrlailnatlon (a), Difficulty (t) , «nd Guessing (<?) 
Parameters for ^' ^ms In t he Mldquarter 2 Stradaptlve Item Pool 



Iton 






b 


(J 


Item 


a 


h 




Item 




a 


b 




a 


StratuB 9 










Strdtun 6 








StratuB 3 












(18 IteHfl) 










(20 Items) 








(17 Items) 












3831 


2.50 


1. 


96 


.06 


3707* 


1.75 


.55 


31 


3634 


1. 


79 




58 


.30 


3690 


2.50 


2. 


36 


.24 


3746* 


1 .59 


,43 


.30 ' 


1739* 


1. 


68 




61 


.35 


3833* 


2. 50 


2. 


65 


. 35 


3806 


1.57 


,48 


.35 


3809 


1. 


27 




61 


.35 


3904 


2.45 


1. 


48 


.28 


3925* 


1.14 


.48 


.35 


3924* 


1. 


13 




79 


.18 


3805 


2.50 


2. 


38 


.35 


3658 


1.24 


.32 


.35 


3672 


1. 


57 


_ 


80 


.15 


3698 


2.11 


2.82 


.35 


3905 


.98 


.35 


.20 


3737* 


1.41 




66 


.34 


3901 


1. 55 


2.62 


.35 


1738* 


1.34 


.40 


.35 


3915 


1.08 




61 


.16 


3835* 


1.21 


2. 


28 


.35 


3605 


1.22 


. 57 


.34 


3640 


1. 


43 




69 


.35 


3620 


2.04 


2. 


97 


.35 


3815 


.95 


.58 


.35 


3906 


.87 




66 


.14 


3697 


1.56 


3. 


00 


.35 


"611 


1 22 


.39 


.32 


3812 




oz 




63 


.13 


3810 




2 


20 


.27 


36'5 


1.21 


.40 


.28 


3682' 


i. 


J J 




72 


.34 


3664 • 


1. li 


1 


60 


.35 


3820 


.92 


.38 


.12 


3637 


1 
1 


nQ 




.73 


.28 


3625 


.98 


1 


66 


.35 


3665 


1.19 


. 54 


.22 


3636 


1 
1 


n A 




63 


.27 


3622 


.95 


2 


53 


.35 


3709* 


1.19 


,30 


.35 


3641 


i. 


ZU 




65 


.22 


384 1* 


.87 


2 


13 


.35 


3724* 


1.14 


.37 


.30 


3711* 


1 


05 




56 


.35 


3651 


.95 


2 


30 


. 35 


3819 


. 76 


.•53 


.35 


3608 


1 


04 




78 


.16 


3728* 


.91 


> 


55 


.35 


3918* 


.6e 


.35 


.23 


3705* 




of 


_ 


58 


.14 


3712* 


.75 


1 


64 


. 10 


3614 


79 


.46 


.35 
























3923* 


.63 


.38 


.11 


Mean (F) 


1 


24 




67 


.24 


Mean (F) 


I, 70 


2 


31 


. 11 


3626 


.65 


. 52 


.25 


Mean*(W) 


1 


2 5 




66 


.25 


Mean*(U) 


1. 58 


2.30 


.32 
































Mean (F) 


1.06 


.46 


.29 


Stratum 2 












Stratum 8 










Mean*(W) 


1. 11 


.44 


.30 


(20 items) 












(18 Items) 


















3735* 


1 


oJ 




94 


.35 


3615 * 


1 . 69 


1 


17 


. 29 


Stratum 5 








3648 


1 


DV 




96 


.33 


3916 


1. 19 


1 


.14 


.35 


{15 items) 








3807 


1 


Ji. 


-.1 


10 


.17 


3673 


1 . 51 


1 


.11 


.31 


3742* 


1.89 


.27 


, 35 


3907 


1 


43 


-1. 


08 


.35 


3804 


. 95 


1 


.42 


. 35 


3745* 


1 . 58 


-^,07 


.20 


3704* 


1 

I 




-1. 


13 


.23 


3733* 


I , ?U 


1 


.40 


. 35 


3720* 


1.45 


, 26 


.29 


3655 


1 
1 






90 


.35 


3719* 


1 . 18 


1 


08 


. 31 


3607 - 


1 . 38 


.09 


.35 


3813 


1 

I 


z u 




97 


.17 


3921* 


. 91 


1 


.23 


.29 


3811 


1.15 


.22 


.35 


3919* 




m 


_ 


98 


.21 


3827 


,87 


1 


.35 


. 35 


3908 


1,15 


.07 


.31 


3680 


1 


J J 


-1 


01 


.16 


3716* 


1 . 14 


1 


.14 


.27 


3649 


1 . 32 


.11 


22 


3808 




99 


-1 


00 


.30 


3642 


1. 11 


1 


.11 


.24 


3632 


1.23 


.27 


,35 


3686 


1 


26 




88 


.29 


3902 


. 71 


1 


.49 


.29 


3718* 


1 ,22 


.16 


.33 


1721* 


1 


.23 


-1 


20 


.22 


3627 


1.03 


1 


.07 


.35 


3629 


i. 11 


-.01 


.35 


3821 




90 




92 


.35 


3681 


03 


1 


.54 


.35 


3732* 


,9f. 


-.01 


.35 


3679 


1 


21 


_ 


94 


.17 


3676 


.89 


1 


.51 


.25 


3633 


.94 


- , G 


.35 


368 5 


1 


. 19 


-1 


01 


.16 


3644 


.88 


1 


.25 


.35 


3609 


.78 


,18 


.35 


3668 




. 97 




87 


.14 


3717* 


.81 


i 


.25 


. 35 


3730* 


75 


.01 


.10 


3684 




. 86 




85 


.14 


3670 


80 


1 


. 1 1 


. 35 


3618 


.64 


-.05 


.00 


3703* 




. 83 


-1 


16 


.21 


3647 


.79 


1 


14 


.35 










3617 




. 79 


-1 


11 


.14 












Mean (T) 


1 ,08 


.09 


29 


3713* 




.75 


-1 


.18 


.11 


I Itr rill \ r f 


I 05 


1 


. 26 


. 12 


Mean*{W) 


1 , 17 


.09 


.28 














Mecin*(W) 


1 . ')5 


1 


.25 


. 32 










Mean (F) 
Mean*{W) 


1 

1 


, 19 

. 1 9 


_ 
-1 


.97 

.01 


.23 
.24 


Stratum ? 










(19 items) 




















MS It f»fT> «i \ 










3744* 


i . 94 


-,15 


. 30 


Stratum 1 












3743* 


J 14 




.68 


32 


1708* 


1 . 62 


-. 20 


.16 


(19 Items) 












36hl 


1 . 90 




.63 


32 


3611 


r, 53 


- , 18 


.35 


3741* 


1 


. 63 


-I 


. 56 


. 35 


3674 


1.72 




.63 


. 26 


1814 


1 26 


-, 12 


.35 


3910 


1 


. 58 


-1 


. 59 


.21 


1909 


1 . 14 




. 77 


. 15 


1901 


1 21 


-,41 


.31 


3692 


1 


.53 


_i 


.28 


.35 


3662 


i . 5m 




.91 


.27 


167 1 


1,51 


- , 14 


, 26 


3825 


1 


.OQ 


-1 


. 38 


. 14 


3654 


I 31 




.84 


.21 


1701 


. 82 


-, 15 


, 35 


1639 


1 


.4/ 


-1 


. 80 


.35 


3669 


1.43 




70 


. 12 


1641 


1 , 40 


50 


.25 


3638 


1 


35 


-1 


. 5^ 


.21 


if)?l 






. 7^ 


. 31 


1914 


. 98 


19 


. 16 


3913 


1 


.31 


-1 


, 11 




39(2 


. 'i 3 




7n 


. 19 


36-:^ J 


1,13 


-,24 


.24 


3837*^ 


1 


.09 


- 1 


. 5^^ 


. 3 


1714* 


. 89 




.96 


, 1'^ 


372 5* 


1 .09 


-, 32 


. 24 


3715* 


1 


.16 


-1 




.26 


3700 


. 84 




83 


. JO 


1710* 


1 , 02 


31 


.30 


3920* 


1 


, 12 


- 1 


, 3^' 


.23 


3659 


1 17 




. 67 


29 


lh51 


83 


-,5l 


,13 


3842* 


1 


,01 


-1 


.55 


,35 


3615 


1.17 




,hh 


13 


3660 


78 


-.19 


, 14 


3695 


1 


.09 


-1 


. 73 


.22 


361 ? 


1 1 J 




.73 


. }3 


1922* 


64 


-.26 


, 30 


1711* 


1 


,05 


-1 


.67 


,35 


361^ 


.Hh 




62 


.23 


lh06 


.71 


-.22 


. 14 


1812 




.99 


-1 


.74 


,12 












1661 




-,17 


,13 


1838* 




.9Q 


-1 


.68 


.35 


Mean (F^ 


1 . i.' 




. 71 


. -9 




,68 


35 


.00 


3613 




,86 


-1 


.74 


,33 


Mean*{W) 


K 31 




7 3 


. )0 


3656 


.61 


-,ll 


. 14 


3683 

Ju T / 




,85 
.81 


-1 
-1 


.31 

, 74 


. 14 

, 1 5 












Mf.in(F) 


1 .01 


- 11 


25 


161(1 




,80 


-1 


. 11 


. 14 












Mt'nn*{V) 


1.08 


-.11 


,26 


Mean (F) 
Mo.in*{W) 


1 
1 


,14 

15 


-1 
-1 


. 54 

.35 


,26 
.2K 






1 


U t I 


" ks Jr 


, • 5i -^^ u'h \ 






1 thf 


p-'nl Wintr 


r ( 


ju.irtcr. 


A! 1 


olhe 






I 






h.-th Fall and Winter quart t-rs. 















Table C 

Item Discrimination (u;. Difficulty (b) , and Guessing 







r d L a mC U c L a 


for Classroom 


Tests Fl 


and F2 ' 




— — 




Fl 








F2 






Item No. 


a 


h ' 


Q Item No. 


a 


b 


a 


3060 


.86 


-1.31 


.29 


3922 


.64 


-.26 ' 


.30 


3067 


1.07 


-.76 


.21 


3904 


2.45 


1.58 


.28 


3065 


1. 17 


-1.66 


.35 


3918 


.66 


.35 


.23 


3056 


. 71 


.89 


.26 


O A1 i 

3921 


.91 


1.23 


.29 


3063 


.91 


1. 51 


.35 


3919 


1. JO 


-.98 


.21 


3073 


1.43 


-1,57 


.31 


J920 


1.12 


-1.34 


.23 


3058 


1.05 


-.43 


. 35 


O A1 O 

3923 


.63 


.38 


.31 


3274 


.85 


-1.05 


.26 


3924 


1.13 


-.79 


.18 


3271 


.95 


1.32 


. 30 


O O At 

3801 


.80 


-.17 


.35 


3055 


1.71 


-.65 


. 24 


3841 


.87 


2.13 


.35 


3072 


1.02 


.65 


. 32 


Q QO C 
JOJO 


.99 


-1.68 


.35 


3057 


1.20 


-1.35 


.26 


o o o o 

3833 


2.50 


2.85 


.35 


3064 


.94 


.86 


.24 


O O O T 


1.09 


-1.59 ■ 


.25 


3069 


.88 


-.01 


.35 


3835 


1.21 


2.28 


.35 


305A 


1.29 


-.93 


.31 


3641 


1.20 


-.65 


.22 


3066 


1.05 


.53 


.31 


3708 


1.62 


-.20 


.16 


3268 ^ 


.97 


-.28 


.18 


3718 


1.22 


.16 


.33 


3267 


1.02 


-1.22 


.23 


3728 


.91 


2 55 


.35 


3272 


1. 06 


-.81 


.35 


3665 


1.19 


.54 


.22- 




. 95 


-1.28 


.22 


3730 


.75 


.01 


.10 


3008 


. 96 


-1.75 


.18 


3719 


1.18 


1.08 


.31 




1 31 


•=29 


.29 


3705 


.87 


-.58 


.14 


3062 


1.47 


.43 


.30 


3713 


.75 


-1.18 


.33 


\J vl X 




1. 57 


.30 


3703 


.83 


-1.16 


.21 




.81 


.47 


.35 


3709 


1, 19 


.30 


.35 




. 99 


2.29 


.35 


3707 


1 75 


.55 


..31 




1. 18 




.32 


'^72i 


1.23 


-1.20 


.22 


3443 


1. 07 


-1.64 


.35 


3717 


.83 


1.25 


.35 


JH JO 


. 70 


.21 


.27 


3715 


1.16 


-1.63 


.26 


3448 


1.40 


.73 


.30 


3716 


1.14 


1.14 


.27 


3435 


.83 


-.61 


.35 


3720 


1.45 


.26 


.29 




1 . 36 


.64 


.32 


3744 


1.94 


- 35 


.30 


3436 


1. 12 


1. 59 


.35 


3745 ^ 


1.58 


-.07 


.20 


'^449 


. 91 


1.26 




J / 4 D 


1.59 


.43 


.30 


'\LL(\ 


1. 52 


2 . 00 


.30 


3711 


1.05 


-.56 


.35 


J / 






.28 


3710 


1.02 


- 33 


.30 


3427 


.92 


1.51 


.26 


3724 


1.14 


.37 


.30 


344^:> 


1.19 


.44 


.34 


3725 


1.09 


-.52 


.24 


^ 3444 


.88 


.78 


.35 


3731 


1 . 03 














3712 


.75 


1.64 


.30 










3704 


1.39 


-1.13 


.23 


Mean 


i.':.4 


.11 


.29 


Mean 


1.17 


.07 


.28 



4!' 



[ 



-33- 



Table D 

Item Discrimination (a). Difficulty (fc), and ';uessing 





(o) Parameters 


for Classroom 


Tests Wl 


and W2 








Wl 








W2 






Item No. 




b 


a 


It^m No. 


a 


h 


a 


328/ 


.85 


-1.28 


. 13 


J / DU 


Q 1 

. 9 3 


1 7Q 




3292 


.68 


1.39 


. 35 


1 n o 

3926 


. 93 


■•1 • jO 


1 f\ 


3219 


1.23 


. ffz 


• 21 


3845 


1 7 1 
1 . / I 


9 A 
. Z 0 


9Q 
. Z^ 


3290 


1.16 


-.57 


• 20 


3763 


1.23 


1 Q 


9ft 
. zo 


3214 


1.12 


.03 


.23 


3762 


1 Q 7 
1.9/ 


— 1 . JO 


1 7 
4 J. / 


3268 


.97 


-.20 


. 18 


OTTO 

Sill 


7 /. 


. O** 


. JJ 


3 '\9 


I.IA 


-1.45 


• 35 


3759 


QQ 
. 99 


1 A 


91 

• Z J. 


3 3 


.96 


-1.30 


. 14 


3768 


1 11 
1.11 


— 1 . J J 


1 7 


3^:^1 


.65 


.52 


. 35 


3756 


1 in 
1 • lU 


9 1 
. Z 1 


9ft 
. zo 


3249 


.91 


-1.69 


. 17 


3749 


1 

1 . U J 




'99 


3083 


1.05 


-.90 


. 13 


1 "7 C T 


1 1 Q 

1 . lo 


— 1 • OU 


1 ft 


3090 


1 8 


-1.65 


. lo 


" o T r c 

37 jj 


1 . U J 


^ 1 9 


1 6 

. X u 


3054 


29 


-.93 


. 31 


3/4/ 


1 11 
i . 1 1 


1 . 0 J 




3084 


1.12 


-1 .06 


1 r 

. 15 


1 "7 C 1 

3753 


0 1 

. y 1 


.J J^. 


1 7 

. X / 


3092 


.9S 


- .65 


. 1 J 


3b 


1 . J 1 


. OH 
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Table E 

Item Discrimination (a). Difficulty (2?), and Guessing (c) Pirameter Estimates for 



Items in the 


Improved 


Conventional 


Tests 


Derived from the Item Pools for 


Tests Wl 


and W2 




Wi 








W2 






.Item Number 


a 


b 


a 


Item Numbed 


a 


b 


c 


3209 


2. 50 


2.29 


.29 


3831 


2.50 
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Table F 

Theoretical Test Information Values for First and 



Second Classroom Tests 


for Fall 


and Winter 


Quarters 


e 

Midpoint 


Test 1 


Teat 2 


Fall 


Winter 


Fall 


Winter 


-1.90 


1.11 


1.89 


1.13 


2.26 


-1.70 


1.46 


2.55 


1.54 


2.86 


-1.50 


1.80 


3.18 


1.97 


3.26 


-1.30 


2.09 
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2.39 
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Table G 

Mean Test Length and Mean Value of Information at Intervals of 6 for 24-Item 



impLu V tiu 




Test and Adaptive Test Rescored at Three Maximum Test Lengths 


for Test Wl 




.Improved 






Adaptive Test 








Conventional 


40 Items 


Max Imum 


30 Items 


Maximum 


20 Items Maximum 


e 


Test 


Mean 


Mean 


Mean 


Mean 


Mean 


Mean 


Midpoint 


Information 


No. Items 


Information 


No. Items 


Information 


No. Items 


Information 


-1.9 


.03 


31 . 3 
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Mean 
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18.1 
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Table H 

Mean Test Length and Mean Value of Informatlonyat Intervals of 6 for 2A-Item 
Improved Conventional Test and Adaptive Test Rescored Three Maximum Test Lengths for Test W2 



Improved 
^ Conventional 
9 Test 
Midpoint Information 



-1.9 


.01 


-1.7 


.03 


-1.5 


.10 


-1.3 


.23 


-1.1 


.51 


-.9 


.98 


-.7 


1.6A 


_ c 


. JH 
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2.90 


-.1 
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.1 
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4.03 


.5 
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4.57 


.9 


4.27 


.1.1 


3.74 


1.3 


3.35 


1.5 


3.26 


1.7 


3.29 


1.9 


3. IS 


Mean 


2.49 



'40 Items Maximum 



Mean 
No. Items 


Mean 
Information 


17.0 


1.99 


34.5 


2.68 


26. 7 


2.91 


24.8 


3.6A 


24. 4 


3.A8 


33. 7 


A. 32 


32. 1 


•i A.A5 


32.0 


A.A9 


25.9 


4.55 


22.9 


3.85 


26.1 


A.6A 


26.0 


A.IA 


29.5 


A.9A 


31.7 


5.60 


29. A 


A. 71 


30.6 


4.12 


30.1 


3.85 


30.2 


A. 11 


31.4 


A. 33 


AO.O 


A. 10 


28.6 


A. 31 



Adaptive Test 



J u 1 L ems 


1 loiv XUlUul 


Mean 


Mo 


No « It ems 


xnx unuau -Luii 


23.5 


1.70 


27.6 


2.A5 


22.2 


2.77 


23.9 


3,A5 


19.0 


2.99 


26.8 


3.73 
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26.3 
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4.72 
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3.16 
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3.9A 
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18. A 
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17.6 


2.95 


17.6 


2.80 


15.5 


2. AS 


20.0 


3.A6 


18.3 


3.A1 
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Table I 

Mean Informaf'on Divided by Mean Number of Items 
at levels of 9 for the Adaptive Tests 



y 


Ad antlve 
Fall 


Test 1 
Winter 


Adaptive 
Fall 


Test 2 
Winter 


-1.9 


.07 


.11 


.07 


.12 


-1.7 


.08 


.08 ' 


.10 


.06 


-1.5 


.10 


.13 


.10 


.11 


-1.3 


.12 


.13 


.11 


.13 


-1.1 


.15 


.13 


.12 


.14 


-.9 


.13 


.16 


.12 


.12 


-.7 


.14 


.17 


.15 


.14 


-.5 


.17 


.16 


. 11 


1 T 
. 1 J 


_ 1 


. ^ ^ 


.21 


.11 


.18 


-.1 


.20 


.19 


.12 


.16 


.1 


.18 


.17 


.12 


17 


.3 


.16 


.18 


.12 


.16 


.5 


.16 


.17 


.11 


.16 


.7 


.16 


.18 


.13 


.16 


.9 


.18 


.17 


.13 


.15 


1.1 


.20 


.18 


.13 


.13 


1.3 


.23 


.19 


.12 


.11 


1.5 


.24 


.16 


.12 


.12 


1.7 


.15 


.18 


.13 


.11 


1.9 


.44 


.32 


.13 • 


.11 


Mean 


.17 


.17 


.12 


.15 
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Arlington, VA 22209 

1 Dr. Robert Young 

Advanced Research Projects Agency 
1400 Wilson Boulevard 
Arlington, VA 22209 

1 Mr. Frede Ick W. Suffa 

Chief, Recruiting and Retention Evaluatio* 
Office of the AsslsUnt Secretary of . 

Defense, M&RA 
Room 3D970, Pentagon 
Washington, DC 20301 

12 Defense Documentation Center 
Cameron SUtion, Bldg. 5 
Alexandria^ VA 22314 
Attn; TC 

1 MinUry Assistant for Human Resources 
Office of the Director cf Defense 

Research & Engineering 
Room 3D129, The Pentagon 
Washington, DC 20301 

1 Director. Management Information 
Systems Office 
OSD. M&RA 

Room 3B917, the Pentagon 
Washington, DC 20301 



2 Aerospac" f'sycholnq. 3ept 
(Code L)) 

Naval Ae-ospacr l^^' Pes L<jb 
Pensacola. f-L 32' ^ 

6 Office of Naval Res^ ^rch 
Code 1021P (ONRL^ 
800 N. Oiiincy Street 
Arlington, VA 22?17 

1 Tng. Analysis & Evaluation Group 
Naval Mq. Equip Cen-Code N«OOT 
Orlar^ , FL 32813 



Marin e C orps^ 

1 Director. Office of Manpower 
Utilization 
Hq» Marine Corps (Code MPU) 
BCB. Building 2009 
Quantico, VA t2134 

1 Dr. A. L. Slafkosky 
Scientific Advisor (Code RD-1 ) 
liQ, U.S. Marine Corps 
Washington. DC 20380 



Other Government 

1 Dr. Lorraine D. Eyde 
Personnel RiD Center 
U.S. Civil Service Commission 
1900 £ Street Nk! 
Was! ington. DC 20415 

1 Dr. William Gorham, Director 
Personnel RiD Center 
U.S. civil Service Commission 
1900 E Street NW 
Washington. DC 2041 & 
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Of . Vtrn Urry 

ftrsonntl UD Ctnttr 

U.S. ClvH $trv1c« CoBinlsslon 

1900 E Street NU 

Weshlngton. DC 20415 

Or, Marsh»31 S. Smith 
Associate Director 
NIE/OPEPA 

N«tiona1 Institute of Education 
Uashlngton. DC 20208 

U.S. Civil Service Conmission 
Federal Office Building 
Chicago Regional Staff Division 
Regional Psychologist 
2?0 S. Dearborn Street 
Chicago. IL 60604 
Attn: C. S. Winlewicz 

Dr. Joseph L. Young, Director 
Memory & Cognitive Processes 
Naticmal Science Foundation 
Washington. DC 20550 

Dr. James M. Ferstl 

Employee Development: Training 

' Technologist 

Bureau of Training 

U.S. Civil Service Commission 

Washington. DC 20415 

William J. McLaurin 
Room 301 

Internal Revenue Service 
2221 Jefferson Davis Hwy* 
Arlington. VA 22202 



llscellaneous 

Dr. John R. Anderson 
Dept. of Psychology 
Yale University 
New Haven. CT 065Z0 

Or, Scarvia B. Anderson 
Educational Testing Service 
Suite 1040 

3445 Peachtree Road NE 
Atlanta. GA 30326 

Professor Earl A. Alluisi 
Code 287 

Dept. of Psychology 
Olcl Dominion University 
Norfolk, VA 23508 

Dr. Daniel Alpert 
Computer-Based Education 
Research Laboratory 
University of Illinois 
Urbana. IL 61801 

M$. Carole A. Bagley 
Applications Analyst 
Minnesota Educational 

Computing Consortium 
1925 Sather Ave. 
Lauderdale. MN 55113 

Mr. Samuel Ball 
Educational Testing Service 
Princeton. NJ 08540 

Dr. Gerald V. Barrett 
University of Akron 
Dept. of Psychology 
Akron. OH 44325 



T Dr. iiobtrt Branson 

lA Tully Bldg. 

Florida State University 

Tallahassee, FL 32306 

1 Dr. John Seeley Brown 
Bolt Beranek and Newinan. 
50 Moulton Street • 
Cambridge. MA 02138 



Inc. 



1 Dr. Victor Bunderson 

Institute for Computer Uses in Education, 
355 EDLC ' 
Brigham Young University 
Provo, UT 84601 

1 Dr. Ronald P. Carver , 
School of Education 
University of Missouri-Kansas City 
5100 Rockh^ll Road 
Kansas City, MO 64110 

1 Jacklyn Caselli ^ 
ERIC Clearinghouse on Information 

Resources 
Stanford University 
School of Education - SCRDT 
Stanford, CA 94305 1 

1 Century Research Corporation 
4113 Lee Highway 
Arlington, VA 22207 

1 

1 Dr. Kenneth £. Clark 

College of Arts i Sciences 
University of Rochester 
R , Campus Station 
Rochester, NY 14627 

1 

1 Dr. Norman Cliff 
Dept. of Psychology 
University of Southern California 
University Park 

Los Angeles, CA 90007 1 

1 Dr. Allan M. Collins 

Bolt Beranek and Newman, Inc. 
50 Moulton Street 

Cambridge. MA 02138 1 

1 Dr. John J. Collins 
Essex Corporation 
201 N. Fairfax St. 
Alexandria. VA 22314 

1 

1 Dr. Rene V. Dawis 
Dept. of Psychology 
University of Minnesota 
Minneapolis, MN 55455 



1 Or. Ruth S. Day 

Center for Advanced Study In 

the Behavioral Sciences 
202 Junipero Serra Blvd. 
Stanford. CA 94305 

1 >. John D. Carroll 
Psychometric Lab 
Davie HaU 01 3A 
Jmversuy of North Carolina 
Chapel Hill . NC 27514 

I Or. Marvin D. Dunnette 

Dept. of Psychology 
University of Minnesota 
Minneapolis. MN 55455 

1 .^IC Facility-Acquisitions 
483J Kuycy Avenue 
Bethesda. MD 20014 



Major I. N. Evonic 
Canadian Forces. Personnel 
Applied Research Unit 
1107 Avenue Road 
Toronto, Ontario, CANADA 

Pr. Richard L. Ferguson 

The American College Testing Program 

P. 0. Box 168 

Iowa City, lA 52240 



Dr. Victor Fields 
Dept. of Psychology 
Montgomery College 
Rockville, HD 20850 

Dr. Edwin A. Fleishman 

Advanced Research Resources Organization 

8555' Sixteenth Street 

Silver Spring, MD 20910 

Dr. Larry Francis 
University of Illinois 
Computer-Based Educational Research Lab 
Champaign, IL 61801 

Dr. John R. Frederiksen 
Bolt Beranek & Newman, Inc. 
50 Moulton Street 
Cambridge, MA 02138 

Dr. Vernon S. Gerlach 
College of Education 
146 Payne Bldg. B 
Arizona State University 
Tempe, AZ 85281 

Dr. Robert Glaser, Co-Director 
Uaiversity of Pittsburgh 
3939 O'Hara Street 
Pittsburgh, PA 15213 

Dr. Rirhard S. Hatch 
Decision Systems Assoc., Inc. 
5640 Nicholson Lane 
Rockville, MD 20852 

Dr. M. D. Havron 
Human Sciences Research, Inc. 
7710 Old Spring House Road 
West Gate Industrial ParK 
McLean, VA 22101 

Dr. Duncan Hansen 
School of Education 
Memphis State University 
Memphis, TN 38118 

CDR Mercer 

CN£T Liaison Officer 

AFHRL/Flying Training Div. 

Williams AFB, AZ 85224 



1 HumRRO/Western Division 
27857 Berwick Drive 
Carmel, CA 93921 
Attn: Library 

1 HumRRO/Columbus Office 

Suite 23, 2601 Cross Country Drive 
Columbus. GA 31906 

1 Dr. Lawrence B. Johnson 

Lawrence Johnson & Associates. Inc. 

Suite 502 

2001 S Street NW 

Washington. DC 20009 
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HontywtlU Inc. 
2600 Rldatway Pkwy. 
• Minneapolis. MN 55413 

1 Dr. Roger A. iCaufman 
203 Dodd Han 
Florida State University 
TallahJsses, FL 32306 

1 Dr. Steven W. Keele 
Dept. of Psychology 
University of Oregon 
Eugene, OR 97403 

1 Or. David Klahr 
Dept. of Psychology 
Carnegie-Mellon University 
Pittsburgh, PA 15213 

ji 

1 Dr.' Ezra S. Krendel 
Wharton School , DH/CC 
Univ. of Pennsylvania 
Philadelphia, PA 19174 

1 Dr. Alma E. Lantz * 
University of Denver 
Denver Research Institute 
Industrial Economics Division 
Denver, CO 80210 

1 Dr. Frederick M. Lord 
Educational Testing Service 
Princeton, NJ 08540 

1 Mr. Brian McNally 

Educational Testing Service 
Princeton, NJ 08540 

1 Dr. Ro R. Mackie 

Human '''s Research, Inc. 
6780 Co f con Drive 
Santa Barbara Research Park 
Goleta. CA S3C17 

1 Mr. Edmond Marks 

• 304 Grange Bldg. 

Pennsylvania State University 
Urnversity Park, PA 16802 

1 Dr. Leo Munday 

Houghton Mifflin Co. 

P. 0. BOX 1970 

Iowa City, lA 52240 

1 Dr. Oonald A. Norman 
Dept. of Psychology C-009 
University of Califorr.la, San D^ego 
la JoUa. CA 92093 

1 Mr. Luigi Petrullo 
2431 N. Edgewood Street 
Arlington, VA 22207 

] ur. Steven M. Pine 
N 660 Elliott ha'»l 
University of Minnesota 
75 East River Road 
Minnea:K)lis, MN 55455 

1 Dr. Lyman W. Porter, Dean 

Graduate School of Administration 
University of California 
Irvine, CA 92717 

1 Or. Diane M. Pair^sey-Klee 
R-< Research & Syste^r. Design 
3947 Ridgemont Drive 
Malibu. CA 90265 
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Bundesminlsterium der Verteldigung 

Postfach 161 

53 Bonn 1 , GERMANY 

1 Dr. Joseph W. Rigney 

University of So. California 
Behavioral Technology Laboratories 
3717 South Grand 
Los Angeles, CA 90007 

1 Dr. Andrew M. Rose 

American Institutes for Research 
1055 Thomas Jefferson St. NW 
Washington, DC 20007 

1 Dr. Leonard L. Rosenbaum, Chairman 
Dept. of Psychology 
Montgomery College 
Rockville, MD 20850 

1 Dr. Mark D. Reckase 

Educational Psychology Dept. 
University of Missouri-Columbia 
12 Hill Hall 
Columbia, MO 65201 

1 Dr. Robert J. Seidel 

Instructional Technology Group, 

HuniRRO 
300 N. Washington St. 
Alexandria, VA 22314 

1 Dr. Richard Snow 
Stanford University 
School of Education 
Stanford, CA 94305 

1 Mr. Dennis J. Sullivan 

c/o Canyon R^sc ch Group, Inc. 
32107 Lindero Canyon Road 
Westiake Village, CA 91360 

1 Mr. Walt W. Tornow 

Control Data Corporation 
Corporate Personnel Research 
P. 0. Box 0 -HQN060 
Minneapolis, MN 55440 

1 Cr. Benton J. Unden^rfOOd 
Dept. of Psychology 
Northwestern University 
Evanston, IL 60201 

1 Dr. Carl R. Vest 

Battelle Memorial Institute 
Washington Operations 
2030 M Street NW 
Washington, ZC 20036 

1 Dr. Keith Wescourt 
Dept. of Psychology 
Stanford University 
Stanford, CA 94305 

1 Dr Earl Hunt 

Dept. of Psycnoioqy 
Umversny of Washington 
Seattle, WA 98105 



1 Dr. Tnoinas G Sticht 

Assoc Director, Basic Skills 
National Institute of Education 
1200 19th Street NW 
Washington, Dt 20208 



Dept. of Psychology 
AdStIn Ptay Hall 304C 
University cf Ttnnewtt 
Knoxvlllti TN. 37916 

1 Dr. Hertdith Crawford 

1 Dr. Nicholls A. Bend 
Dept. of Psychology 
Sacramento State College 
6000 Jay Street 
Sacramento » CA 95819 

1 Dr. James Greeno 
Learning R&D Center 
University of Pittsburgh 
3939 O'Hara Strwt 
Pittsburgli, PA 15213 

1. Dr. Frederick Hayes-Roth 
The Rand Corporation 
1700 Main Street 
Santa Kon1ca> CA 90406 

1 Dr. Robert Sternberg 

Dept. of Psychol ogy> ^ ^ 

Yale University* 

Box llA, Yale Station 

New Haven, CT 06520 

1 Dr, Walter Schneider 
Dept. of Psychology 
University of Illinois 
Champaign, IL 61820 

1 Dr, Richard B. Mlllward 
Dept. of Psychology 
Hunter Lab 
Brown University 
Providence, RI 82912 

1 American Institute for Research 
1055 Thomas Jefferson St., N.W. 
Washington, D.C. 20007 

1 Applied Psychological Services 
Science Center 
404 East Lancaster Ave. 
Wayne. PA 19087 , 

1 Bolt, Beranek, and Newman, Inc. 
50 Moulton St. ' 
ATTN: Library 

Cambridge, MA 02138 

1 Dunlap Associates, Inc. 
One Parkland Drive 
ATTN- Library 
Darien, Cr 0G820 

1 Educational Testing Service 
ATTN: Library 
Rosedale Road 
Princeton, NJ 08540 

1 HumRRO/Eastern Division 
300 North Washington St. 
ATTN: Library 
Alexandria, VA 22314 



1 Rockwell International 

ln<i Angeles International Airport 
ATTN- B-1 Div. TIC Dept. 299 
Los Angeles, CA 90009 
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0«ve1 opMtnt Corpori 1 1 qn 
ATTN: Technical Info Ctr 
Han Drop ^41-41 
2S00 Colorado Ave. 
S^nta Monica, CA 90406 

Advanced Research Resources Organization 
85SS 16th Street 
Silver Spring, MO 20910 

y Colorado State University 
Department of Psychology 
Ft. Co U ins, CO 80521 / 

1 University of Denver 
Department of Psychology 
'2030 StJuth York 
Denver, CO 80210 

Florida State University 
C^l Center 
Tul Ty Building' 
Tallahassee, FL 32306 

\ Dept. of Behavioral Sciences 
ilnlversity of Chicago 
B848 S. University Ave. 
Chiiago, IL 60601 



1 Illinois State Univers'ty 
ATTN: Document Librarian 
Normal. IL 61761 

1 Oociwents Division 
University of Illinois 
ATTN: Library 
Urbana. IL 47907 

1 Department of Psychology 
Purdue University 
Lafayette. IN 47907 

1 Department of Psychology 
University Houston 
Houston. TX 77004 

Dr Danile Alpert 
Computer- ocsed Education 
Research Laboratory 
University of Illinois 
Urbana, IL 61820 

Dr. Ernest J. Anastasio 
Associate Director 
Data Analysis Research Division 
Educational Testing Service 
Princeton. NJ 08540 

Mr. Avron B Barr 
.19 Ventura Hall 
I.M.S.S,S. 

Stanford University 
Stanford. CA 94305 

Dr. Arthur S. Blaiwes 
Naval Training Equipment Center 
Code N 215 
Orlando. FL 32813 

Dr. Victor Bunderson 
Brigham Young University 
Institute for Computer Uses m Ed 
W 164 STAD 
Provo. UT 84001 

Dr. PoUy C-arpenter-Huf fman 
The Rand Corporation 
1700 Main Street 
Santa Monica, CA 90401 



1 Dr. Jeanne S. Chall 

Graduate School ^ Education 
Harvard University 
Roy E. Larsen Hall 
Appian Way 

Cambridge. MA 02138 

1 Dr. Irene Clements 
Dept. of Home Economics 
Box 3516 
U5A0 

Chickasha. OK 73018 

1 Eugene W. Dalzeli . Jr. 
Adv Tng Dev Proy 
General Electric ^o, 
100 Platers Ave. 
Pittsfield. MA 01201 

1 Dr. John Eschenbrenner 
McDonnell Douglas 
P.O. Box 30204 
Lowry AFB. CO 80230 

1 Mr, Frederick Finch 
CTB/McGraw Hill 
Del Monte Research Park 
Monterey. CA 93940 

1 Dr. Wally Fuerzig 

Dept. of E^^uc. Technology 
BBN 

50 Moulton St. 
Cambridge. MA 02138 

1 Mr. Robert M. Gagne 
Instr. Design 
College of Education 
FloriHa State Univers'ity 
Tallahassee. FL 32306 

1 Mr. R. N. Hale 2-56220 
VouPht Aeronautics Division 
P.O. Box 5907 
Dallas, TX 75222 

] Dr. Duncan Hansen 
School of Education 
Memphis State University 
Memphis, TN 38118 

] M. R. C. Houston 

American Airlines, Inc. 
Greater Southwest Intl Airport 
Fort Worth, TX 76125 

1 Dr. Richard E. Hurlock 
PLATO IV Evaluation 
Code 9304 

Navy Personnel Research & 
Developnent Center 
San Diego. CA 92152 

2 Dr. Kirk Johnson 

NPRDC Branch Office. Memphis 
Building S-39 
Millington. TN 38504 

1 Dr Gregory A. Kimble 
Dept. of Psychology 
University of Colorado 
Boulder, CO 80302 

I Dr. Georqe R Klare 
Dept. of Psycholoqy 
Ohio University 
35 Park Place 
Athens, OH 45701 



2 Mr. C. S. Hfcely 

Manager, Training (Code 54-40) 
Douglas Aircraft Company 
3855 Ladewood Blvd. 
Long Beach, CA 90846 

1 George W. Powell , M.D. FACP 
Gen Dynumics-Corvair Division 
^.0. Box 80877 - NA 130-20 
San Diego, CA 92138 

1 Dr. Tillman J. Ragan 
College of Education 
University of Oklahoma 
^ Norwan, OK 73069 

I Mr. Joseph W. Rigney 
Behavioral Technology Labs 
University of Southern California 
3717 W. Grand Avenue 
Lo- Angeles, CA 90007 

1 Ga^y L. Slinmer, EdD 

Manager, Postal Employee Dev Ctr 
Main Post Office 
5th & Kansas 
Topeka, KS 66603 

1 Mr. Willia.i Stobie 

McDonnell Douglas Corporatidn 

P.O. Box 516 

St. Louis. MO 63166 

1 Dr. Laurence M. Stolurow, 
State University of New York 
Stony Brook, NY 11794 

1 Dr. Calvin -W. Taylor 
Department of Psycholoqy 
University of Utah * 
Salt Lake City, UT 84112 

1 Dr. Eric McWilliams 
Program Manager 
Technology i Systems. IIE 
National Science 'Foundation 
Washington, D.C. 20550 

1 Dr. Arthur W. Melton 
Human 'Performance Center 
University of Michigan 
Ann Arbor. MI 48104 

> Ethel M. Nance. Manager 

Postal Employee Development Ctr 
Room 4029 

Main Post Office Bui id mo 
Cleveland. OH 4410> 
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